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IMAGE PROCESSING 



The present invention is concerned with image processing. 

Image segmentation technology has immediate application to compression where image 
regions can be identified and economically coded for storage or transmission. Normally 
segmentation identifies regions that are uniform and homogeneous with respect to some 
characteristic such as colour or texture. It is essential that the processing does not 
damage the perceptual quality of the image and so the segmentation must not interfere 
with those locations in an image that possess meaning and attract visual attention. 

There is a huge literature on the subject of segmentation [5] and a wealth of results on 
specific applications: 

• Feature based methods [2,8,9,10] rely upon pre-specified measurements such as 
colour and texture, that map pixels into clusters representing different regions in the 
image. Currently there is no way of knowing" the number of clusters that may be present in 
an image or whether they correspond to real objects. Furthermore the features are often 
image dependent and it is unclear how to select such features to obtain satisfactory 
performance. 

• Region based approaches [3, 4] have been developed that take account of the 
spatial location of pixels to obtain region compactness. However, such methods have an 
inherent dependence on the initial conditions and the order in which pixels are examined. 

• Many techniques incorporate edge detection [1] to define the boundaries of the 
regions they contain. These approaches do not work well where the edges are numerous 
or not well defined. In addition they are sensitive to noise and it is often difficult to ensure 
that contours or boundaries are closed. 

• Colour histograms [7] rely upon the presence of peaks of colour frequencies and 
therefore do not work well on images without such peaks. Furthermore no account is 
taken of spatial relationships and region contiguity cannot be guaranteed. 

[1] X Jie and S Peng-fei, "Natural color image segmentation," ICIP 2003. 

[2] . Chen, T N Pappas, A Mojsilovic and B E Rogoyvitz, "Image segmentation by spatially 
adaptive color and texture filters," ICIP 2003. 
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• [3] H Xue, T Geraud and A Duret-Lutz, "Multi-band segmentation using morphological 
clustering and fusion - application to color image segmentation," ICIP 2003. 

[4] P Lambert and H Grecu, "A quick and course color image segmentation/' ICIP 2003. 

[5] H D Cheng, X H Jiang, Y Sun and J Wang, "Color image segmentation : advances 
and prospects," Pattern Recognition, vol 34, no 9, pp 2259-2281, 2001. 
[6] -F W M Stentiford, "An estimator for visual attention through competitive novelty 
with application to image compression," Picture Coding Symposium, Seoul, 24-27 April, 
200 1 . 

[7] E Littmann and H Ritter, "Adaptive color segmentation - a comparison of neural 

and statistical methods," IEEE Trans. Neural Network, 8, (1), pp 175-185, 1997. 

[8] L Itti, C Koch and E Niebur, "A model of saliency-based visual attention for rapid 

scene analysis," IEEE Trans on PAMI, vol 20, no 1 1, pp 1254-1259, Nov 1998. 

[9] W Osberger and A J Maeder, "Automatic identification of perceptually important 

regions in an image," Proc 14 th IEEE Int. conf on Pattern Recognition, pp 701-704, August 

1998. 

[10] W Osberger, US Patent Application no 2002/0126891, "Visual Attention Model," 
Sept 12 2002. 

According to the present invention there is provided a method of processing a digitally 
coded image in which picture elements are each represented by a colour value, comprising, 
for each of a plurality of said picture elements: 

(a) performing a plurality of comparisons, each comparison comprising comparing a first 
picture element group which comprises the picture element under consideration and at 

• least one further picture element in the vicinity thereof with a second picture element group 
which comprises a base picture element and at least one further picture element, the 
number of picture elements in the second group being the same as the number of picture 
elements in the first group and the position of the or each further element of the second 
group relative to the base picture element of the second group being the same as the 

* position of the or a respective further element of the first group relative to the picture 
element under consideration, wherein each comparison determines whether the two groups 
match in the sense that they meet a criterion of similarity; and 

(b) when at least one comparison results in a match, computing a replacement colour value 
for the picture element under consideration, the replacement colour value being a function 



of the colour value for the base picture element of the or each second group for a match 
was obtained. 

Preferably, the method includes identifying picture elements which meet a criterion of 
distinctiveness, and computing a replacement colour value only for picture elements not 
meeting the distinctiveness criterion. 

Other, preferred, aspects of the invention are defined in the claims. 

Some embodiments of the invention will now be described, by way of example, with 
reference to the accompanying drawings, in which: 

Figure 1 is a block diagram of an apparatus for performing the invention; 

Figure 2 is a flowchart of the steps to be performed by the apparatus of Figure 1 in 
accordance with one embodiment of the invention; 

Figure 3 is a similar flowchart for a second embodiment of the invention; 

Figure 4 is a similar flowchart for a third embodiment of the invention; and 

Figures 5 to 8 illustrate the effects of this processing on some sample images. ^ 

Figure 1 shows an apparatus consisting of a general purpose computer programmed to 
perform image analysis according to a first embodiment of the invention. It has a bus 1, to 
which are connected a central processing unit 2, a visual display 3, a keyboard 4, a 
scanner 5 (or other input device, not shown) for input of images, and a memory 6. 

In the memory 6 are stored an operating system 601, a program 602 for performing the 
image analysis, and storage areas 603, 604 for storing an image to be processed and a 
processed image, respectively. Each image is stored as a two-dimensional array of values, 
each value representing the brightness and/or colour of a picture element within the array. 

One embodiment of the invention is shown in the flowchart of Figure 2. An image to be 
processed is stored as an array C of pixels where the position of a pixel is expressed in 
Cartesian co-ordinates e.g. (x„ x 2 ) or as a vector (in bold type) e.g. x= (x lf x 2 ). The colour 
of a pixel at x is stored as a vector C(x) consisting of three components. In these 
examples r,g,b components are used but other colour spaces could be employed. In a 
monochrome system C would have only one (luminance) component. The results of this 
process are to be stored in a similar array C 0UT . 

The process is iterative and starts at Step 100 which simply indicates that one begins with 
one pixel x and repeats for each other pixel (the order which this is performed is not 



significant), terminating at Step 102 when ail have been processed. However it is not 
essential to process all pixels: some may be deliberately excluded, for reasons that will be 
discussed presently. 

' In Step 104, a comparison count I is set to zero, a match count M is set to 1, and a colour 
vector V is set to the colour at x. V has three components which take values according to 
the colour space employed e.g. (r,g,b). 

Step 106 

n (typically 3) random pixels at x'j = (x' /7 , x' l2 ) are selected in the neighbourhood of x 
where 

| x'ij- x f \ < rj for all j and r, defines the size of a rectangular neighbourhood. A typical 
value for rj would be 2 for a 640 x 416 image. 

Step 108 - 

A random pixel at y = (y„y 2 ) is then randomly selected elsewhere in the image and (Step 
110) the comparison count I incremented. This pixel is selected to be > ^ from the image 
boundary to avoid edge effects. If desired, the choice of y could be limited to lie within a 
maximum distance from x. If, at Step 1 1 2, the value of I does not exceed the value of a 
threshold L (typical values are 10 - 100) a test for a match between the neighbourhoods of 
x and y is carried out. 

Step 114 

Let the colour of the pixel at x be C(x) = (C^x), C 2 (x), C 3 (x)) = (r x , g x , b x ) • 

Then the neighbourhoods match if each of the pixels x, x' { (that is, the pixel under 
consideration and its n neighbouring pixels) matches the corresponding pixel at y, y'„ 
where the positions of y', relative to y are the same as those of x'i relative to x. That is to 
say: 

x - x', = y - y'j for all i = 1 , n. 

where pixels at x and y are deemed to match if 

|C,(x) - C,(y) | < dj for all j 

and similarly for x', and yV 

iCjfx'i) - qfy'j) | < dj for all j and all i 

where dj is a threshold that determines whether colour component j is sufficiently, different 
to constitute a pixel mismatch. In the tests described below, the colour components were 



represented on a scale of 0 to 255 and a single value of dj = 80 was used. In general dj 
may be dependent upon x. For example, it may be preferred to model attention so that 
less emphasis is given to darker regions by increasing dj in these areas. 

If a match is found then at Step 1 1 6 the counter M is incremented and the values of the 
colour components at y are added to V, 

V = V + C.(y) 

Following a match the process returns to Step 106 of selecting a fresh neighbourhood 
around x containing n random pixels, whereas if no match is found it returns to Step 108 
to select a new y without changing the pixel neighbourhood. 

If at Step 1 12 the value of I exceeds the threshold L, the colour of the pixel at x = (x u x 2 ) 
in the transformed image is given (Step 118) the average value of the colours of the M 
pixels found to have matching neighbourhoods i.e. 

Cout(x) = V / M This process is repeated from Step 100 until all pixels in the image have 
been dealt with. The resulting transformed image possesses a much reduced spread o| 
colours but also contains small levels of noise arising from the random nature of the 
algorithm. This noise is simply removed by applying a standard smoothing algorithm. Iq. 
this embodiment a pixel is assigned the average colour of the pixels in the surrounding 3 % 
3 window. 

The algorithm shown in Figure 2 processes all pixels x, and all will have their colours 
altered except in the case of pixels whose neighbourhoods are so dissimilar to the rest of 
the image that no matches are found. In that the process necessarily involves a loss of 
information, we prefer to identify important parts of the image and exclude these. Thus 
the embodiment of Figure 3 excludes regions of interest from the filtering process. In 
Figure 3, those steps which are identical to those of Figure 2 are given the same reference 
numerals. 

The process begins at Step 130 with the generation of a saliepcy map consisting of an 
array of attention scores Scores^ ,x 2 ) using the method described in our international 
patent application WO01/61648 (Agents ref. A25904). Other methods of generating 
saliency maps may also be used although their performance may not always be best suited 
to this application [8,9,10]. The values of Scores(x lf x 2 ) assigned to each pixel at x = 
ix u x 2 ) or a subset of pixels in an image reflect the level of attention at that location. 

A value is given, to the variable T, typically 0.9, which sets a threshold on Scores(x u x 2 ) and 
determines whether the colour of the pixel at x is to be transformed or not where 



Threshold = 7"* (max - min) + min 

and max = Max (Scores (x l9 x 2 )) min = Min(Scores(x u x 2 )) . However, other means of 

calculating the value of Threshold may be used some of which can be dependent upon x. 

If at Step 1 32 the value of Scores(x / ,x 2 ) is greater than Threshold, the pixel in the original 
image at x is, at Step 134, copied unchanged into the transformed image array 
Outputimage(x ? ,x 2 ). This pixel represents a point of high attention in the image and will not 
be altered by this process. 

The remainder of the process is as previously described: note however that in the 
smoothing algorithm, the colour value is replaced by the smoothed value only for those 
pixels whose attention scores are less than the value of Threshold. 

Another embodiment is shown in Figure 4 in which attention scores are not computed 
beforehand. Instead, when at Step 1 12 the comparison count I exceeds the threshold L, a 
test is performed at Step 1 50 to determine whether the match count M is greater than a 
threshold mt. If so, then, as before, the colour of the pixel at (x lf x 2 ) in the transformed 
image is given, at Step 1 18,- the average value of the colours of the M pixels found to 
have matching neighbourhoods i.e. 

V / M 

If, however, M is less than or equal to mt, the pixel in the original image at x is, at Step 
152, copied unchanged into the transformed image array Outputimage(x r ,x 2 ). This means 
that pixels representing areas of high attention will be unlikely to be altered because only 
low values of M will be obtained in these image regions. 

The degree of filtering that is applied to the image may be controlled by selecting the value 
of the thresholds dj. Alternatively, or in addition, the filtering process can if desired be 
repeated: as shown at Step 170. The transformed image may be reloaded whilst (in the 
case of Figure 4) retaining the original attention scores Scores^, x 2 ) and the whole process 
repeated to obtain successive transformations and greater suppression of background 
information. 

Note that where random selection is called for, pseudo-random selection may be used 
instead. 

Once filtering is complete, the transformed image may if desired be encoded in JPEG 
format (or any other compression algorithm) as shown at Step 180. The reduction in 



information contained in regions of low interest enables higher compression performances 
to be attained than on the original image. 

The results of applying the algorithm of Figure 4 to a football source are shown in Figure 5, 
which shows, from left to right, the original image (GIF format), the image after JPEG 
coding, and the image after filtering followed by JPEG CODING. Histograms of the 
distribution of hue values in the range 1-100 are also shown. It is found that the filtering 
reduces the compressed image file size from 13719 bytes to 10853 bytes. 
Typically two iterations of pixel colour replacement and smoothing are sufficient, but this 
can be extended depending upon the colour reduction required. 

Figure 6 illustrates how background information may be substantially removed whilst 
preserving important features of the image such as the boat and the mountain outline. The 
original JPEG encoding (Figure 6a) occupies 13361 bytes which is reduced to 10158 bytes 
after processing once and JPEG encoding the transformed version (Figure 6b). The 
Outputimage is reprocessed using the same VA scores and obtains a file size of 8881 
bytes (Figure 6c). A further iteration obtains a size of 8317 bytes {Figure 6d). 

•5* 

This method may be applied with advantage to images containing artefacts (such as JPEG 
blocking effects). The re-assignment of colours to background regions tends to remove 
artefacts which normally possess some similarity to their surroundings (See Figure 7, 
where the original image is shown on the left: on the right is shown the image obtained 
following processing with this method and subsequent >e-coding using JPEG). However, 
artefacts which are very obtrusive and interfere with the main subject material will not be 
removed. 

A further application of the method is the enhancement of figure-ground or the removal of 
background distractions for improved recognisability. This application is illustrated in 
Figure 8 in which the background is almost completely replaced with a constant colour and 
the image of the dog is much more prominent. The method could therefore be applied to 
the processing of images displayed in a digital viewfinder in a camera where the 
enhancement of subject material will assist photographers to compose their pictures. 

Essential visual information is retained in the transformed images whilst reducing the 
variability of colours in unimportant areas. The transformed image thereby become much 
easier to segment using conventional algorithms because there are fewer colour boundaries 
to negotiate and shape outlines are more distinct. This means that this method will 
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erihance the performance of many conventional algorithms that seek to partition images 
into separate and meaningful homogeneous regions for whatever purpose. 
In the embodiments we have described, the replacement colour value used is the average 
of the original value and those of all the pixels which, it was fount to match (although in 
fact it is not essential that the original value be included). Although it practice this does 
not necessarily result in a reduction in the number of different colour values in the image, 
nevertheless it results in a reduction in the colour variability and hence - as has been 
demonstrated - increases the scope for compression and/or reduces the perception of 
artefacts in the image. Other replacement strategies may be adopted instead. For 
example, having obtained the average, the replacement could be chosen to be that one of a 
more limited (i.e. more coarsely quantised) range of colours to which the average is 
closest. Or the match results could be used to identify groups of pixels which could then 
all be assigned the same colour value. 

These embodiments assume that low level segmentation algorithms should not be applied 
to those areas in an image that merit high visual attention. Such regions are naturally 
anomalous and contain a high density of meaningful information for an observer. This 
means that any attempt to segment these areas is likely to be arbitrary because there is 
little or no information in the surrounding regions or elsewhere in the image that can be 
usefully extrapolated. On the other hand less significant parts of the image that are more 
extensive can justifiably be transformed using quite primitive and low level algorithms. 
Paradoxically distinctive object edges in an image attract high attention and therefore are 
not subjected to alteration in this approach. In fact the edges of objects at the pixel level 
in real images are extremely complex and diverse and would need specifically tailored 
algorithms to be sure of a correct result in each case. 

The second and third embodiments of the invention offer an approach to colour 
compression that makes use of a visual attention algorithm to determine visually important 
areas in the image which are not to be transformed. This approach therefore possesses 
the significant advantage that the process of assigning region identities does not have to 
address the difficult problem of defining edges which normally hold the highest density of 
meaningful information. Non-attentive regions are transformed according to parameters 
derived from the same VA algorithm which indicates those regions sharing properties with 
many other parts of the image. The visual attention algorithm does not rely upon the pre- 
selection of features and hence has application to a greater range of images than standard 
feature based methods which tend to be tailored to work on categories of images most 



suited to the selected feature measurements. Pixels in the regions subject to 
transformation are assigned an average colour and increased compression obtained through 
JPEG encoding or any other compression standard. Compression is applied to the least 
attentive regions of the image and therefore is unlikely to affect the perceptual quality of 
the overall image. The algorithm may be iteratively applied to the transformed images to 
obtain further compression at the expense of more background detail. Previous 
applications of visual attention models to compression [6] used a coarser quantisation for 
the regions of low importance, but this can lead to distracting blocking artefacts in those 
areas. 
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Claims 

1 . A method of processing a digitally coded image in which picture elements are each 
represented by a colour value, comprising, for each of a plurality of said picture elements: 

(a) performing a plurality of comparisons, each comparison comprising comparing a first 
picture element group which comprises the picture element under consideration and at 
least one further picture element in the vicinity thereof with a second picture element group 
which comprises a base picture element and at least one further picture element, the 
number of picture elements in the second group being the same as the number of picture 
elements in the first group and the position of the or each further element of the second 
group relative to the base picture element of the second group being the same as the 
position of the or a respective further element of the first group relative to the picture 
element under consideration, wherein each comparison determines whether the two groups 
match in the sense that they meet a criterion of similarity; and 

(b) when at least one comparison results in a match, computing a replacement colour value 
for the picture element under consideration, the replacement colour value being a function 
of the colour value for the base picture element of the or each second group for a match 
was obtained. 

2. A method according to claim 1 including identifying picture elements which meet a 
criterion of distinctiveness, and computing a replacement colour value only for picture 
elements not meeting the distinctiveness criterion. 

3. A method according to claim 2 wherein the step of identifying picture elements 
which meet a criterion of distinctiveness is performed in advance, and said comparisons are 
performed only for picture elements not meeting the distinctiveness criterion. 

4.. A method according to claim 2 wherein the criterion of distinctiveness for a picture 
element is that the number of matches obtained for that picture element exceeds a 
threshold. 
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5. A method according to any one of the preceding claims wherein the selection of the 
or each further picture element of the first group is selected in a random or pseudo-random 
manner. 

6. A method according to any one of the claims 1 to 4 wherein the selection of the or 
each further picture element of the first group is selected in a random or pseudo-random 
manner from picture elements lying within a predetermined distance of the element under 
consideration. 

7. A method according to any one of the preceding claims wherein the further 
elements are selected afresh following a match. 

8. A method according to any one of the preceding claims wherein the selection of the 
base picture element of the second group is selected in a random or pseudo-random 
manner. 

9. A method according to any one of claims 1 to 7 wherein the selection of the base 
picture element of the second group is selected in a random or pseudo-random manner 
from picture elements lying within a predetermined distance of the element under 
consideration. 

10. A method according to any one of the preceding claims wherein the base picture 
element for the second group is selected afresh for each comparison. 

11. A method according to any one of the preceding claims wherein the image is a 
monochrome image and the colour value is a single, luminance component. 

12. A method according to any one of the claims 1 to 10 wherein the image is a colour 
image and the colour value has three components. 




13. A method according to claim 11 or 12 wherein the match criterion is that no 
component of any picture element of the first group differs from the corresponding 
component of the spatially corresponding element of the second group by more than a 
threshold amount. 

14. A method according to any one of the preceding claims wherein the replacement 
colour value for a pixel is a function also of its existing colour value. 

15. A method according to claim 14 wherein the replacement colour value is the 
average of the colour value for the picture element under consideration and the colour 
value for the base picture element of the or each second group for a match was obtained. 

16. A method according to claim 14 wherein the replacement colour value is that one of 
a predetermined set of colour values which is closest to the average of the colour value for 
the picture element under consideration and the colour value for the base picture element 
of the or each second group for a match was obtained. 

17. A method according to any one of the preceding claims including the step of 
processing the processed image again using the method of claim 1 . 

18. A method according to any one of the preceding claims including the step of 
applying spatial filtering to the processed image. 

19. A method according to any one of the preceding claims including the step of 
encoding the processed image using a compression algorithm. 
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This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 



IH COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

^REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 
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